Shared Virtual Memory Clusters with Next-Generation Interconnection Networks and Wide Compute Nodes

Recently much effort has been spent on providing a shared address space abstraction on clusters of small–scale symmetric multiprocessors. However, advances in technology will soon make it possible to construct these clusters with larger–scale cc-NUMA nodes, connected with non-coherent networks that offer latencies and bandwidth comparable to interconnection networks used in hardware cache–coherent systems. The shared memory abstraction can be provided on these systems in software across nodes and in hardware within nodes. In this work we investigate this approach to building future software shared memory clusters. We use an existing, large–scale hardware cache–coherent system with 64 processors to emulate a future cluster. We present results for both 32and 64-processor system configurations. We quantify the effects of faster interconnects and wide, NUMA nodes on system design and identify the areas where more research is required for future SVM clusters. We find that current SVM protocols can only partially take advantage of faster interconnects and they need to be adjusted to the new system features. In particular, unlike in today’s clusters that employ SMP nodes, improving intra–node synchronization and data placement are key issues for future clusters. Data wait time and synchronization costs are not major issues, when not affected by the cost of page invalidations.